General plugin mechanism #45355

weishengying · 2022-08-23T12:01:30Z

PR types

New features

PR changes

Others

Describe

通用 Plugin 以及自定义Op Plugin 加载机制

方案要点：

当前paddle trt 工作流程图如下：

这部分的主要功能就是利用 xxOpConverter 对象将Paddle Op 转换为 TRT Layer。新方案新增两个 converter（creater），能够使以下算子进入TRT。
第一类Op：该Op被通用 Plugin 所支持；
第二类Op：用户自定义Op

新增两个 converter：generic_and_custom_plugin_creater.cc
generic_plugin_creater ：该 converter 为一些 Op 创建通用Plugin；
custom_plugin_creater ：该 converter 为自定义 Op 创建用户自定义的Plugin；

调整了OpTell；

对于 paddle 已有的大量 op converter，加上新增的两个 converter，已有三种converter。对于一个 paddle op 需要使用哪种 converter，由 OpTeller 判断。

之前 OpTeller 的功能是判断一个 Paddle Op ， TRT 支持与否，其中大量的 if 判断放在了 op_converter.h文件中，这不太合理，op_converter.h 只负责根据Op名字，找到对应的 converter，而能不能转的 if 等边界条件判断，应当移入 op_tell.cc 中。

由于现在存在三种 converter，因此 OpTeller 不仅要告知一个 Paddle Op ， TRT 支持与否，还需要告知使用哪一种 converter。

enum class OpConverterType {
  Default = 0,
  GenericPluginCreater,
  CustomPluginCreater
};

Default 表示使用框架内部已有的 xx_op_converter。

测试

1. 测试 custom_plugin_creater

通过在 dynamic_shape_infermeta.cc文件中增加符号化的 shape 推导函数，使得对应的 Op 被通用 plugin 所支持。
在已有的auto_scan的单测中便能够测试通用plugin对该 Op的支持情况。
如在 dynamic_shape_infermeta.cc 中增加了 gather_nd op 的符号化shape推导函数，在 test_trt_convert_gather_nd.py单测中就会使用通用 plugin。

2. 测试 custom_plugin_creater

test_custom_plugin_creater.cc
test_custom_op_plugin.cc

测试 custom_plugin_creater 的功能：能否加载用户自定义 Op 的 Plugin。主要测试 custom_plugin_creater 能否正确加载到用户自定义Op，获取Op的各种属性信息，以及从 IPluinRegistry 中得到正确的 plugin_creator，将自定义Op的属性正确传给
plugin_creator，然后成功创建自定义Op的plugin。

规定：
用户自定义Op的静态shape plugin，以名称 “_paddle_trt_plugin”结尾; 动态shape plugin 以 “_paddle_trt_dynamic_plugin”结尾。
在 test_custom_op_plugin.cc 文件下为 custom_op 定义了一个静态shape plugin 和动态 shape plugin。

这两个plugin 直接继承 public nvinfer1::IPluginV2 ，public nvinfer1::IPluginV2DynamicExt，

两个plugin 都没有具体的计算逻辑。主要实现的接口为：

nvinfer1::IPluginV2* createPlugin(
      const char* name,
      const nvinfer1::PluginFieldCollection* fc) noexcept override;

该 api 负责接收自定义Op的属性信息，然后创建 plugin 对象。

paddle-bot · 2022-08-23T12:01:34Z

你的PR提交成功，感谢你对开源项目的贡献!
请关注后续CI自动化测试结果，详情请参考Paddle-CI手册。
Your PR has been submitted. Thanks for your contribution!
Please wait for the result of CI firstly. See Paddle CI Manual for details.

paddle-bot · 2022-08-23T12:01:36Z

✅ This PR's description meets the template requirements!
Please wait for other CI results.

qingqing01

add unit testing

qingqing01 · 2022-08-25T04:11:38Z

paddle/fluid/inference/tensorrt/convert/generic_and_custom_plugin_creater.cc

+        plugindata.data = &value;
+      } else {
+        CHECK(false) << "not incompleted";
+      }


Why not support float, and other dtype ?

已补充完整

qingqing01 · 2022-08-25T04:14:52Z

paddle/fluid/inference/tensorrt/convert/generic_and_custom_plugin_creater.cc

+      plugindatas.push_back(plugindata);
+    }
+
+    nvinfer1::PluginFieldCollection pluginFC{(int32_t)plugindatas.size(),


“pluginFC” should lowercase with underscore, see https://google.github.io/styleguide/cppguide.html#Variable_Names

qingqing01 · 2022-08-25T04:15:50Z

paddle/fluid/inference/tensorrt/convert/generic_and_custom_plugin_creater.cc

+    }
+
+    auto creator =
+        GetPluginRegistry()->getPluginCreator(op_desc.Type().c_str(), "1");


add comments for why setting plugin version to “1”

qingqing01 · 2022-08-25T04:25:14Z

paddle/fluid/inference/tensorrt/convert/generic_and_custom_plugin_creater.cc

+        auto *var = block_desc.FindVar(arg_name);
+        PADDLE_ENFORCE_NOT_NULL(
+            var,
+            platform::errors::NotFound("no variable called %s in block.",


no variable called -> There is no variable called

qingqing01 · 2022-08-25T04:27:36Z

paddle/fluid/inference/tensorrt/convert/generic_and_custom_plugin_creater.cc

+class OpDesc;
+}  // namespace proto
+}  // namespace framework
+}  // namespace paddle


Remove line 21 to line 28

qingqing01 · 2022-08-25T04:32:27Z

paddle/fluid/inference/tensorrt/dynamic_shape_infermeta_factory.h

+                            const nvinfer1::DimsExprs* inputs,
+                            int nb_inputs,
+                            nvinfer1::IExprBuilder& expr_builder,  // NOLINT
+                            const framework::OpDesc& op_desc_);


op_desc_ -> op_desc

qingqing01 · 2022-08-25T04:34:12Z

paddle/fluid/inference/tensorrt/op_teller.cc

+        return false;
+      }
+      return true;
+    }


GenericPlugin对op的输入、输出类型也有限制，这里是否要增加判断

根据 op 做具体判断，后续继续添加通用plugin支持的op时，根据op的具体情况做判断

qingqing01 · 2022-08-25T04:38:10Z

paddle/fluid/inference/tensorrt/plugin/generic_plugin.cu

+    auto* attr_ptr = attr_reader.GetAttr(attr_name);
+    switch (attr_defs[k].type_index) {
+      case phi::AttributeType::SCALAR:
+        if (attr_ptr) {


这里 if(attr_prt) 判断是否需要提前到49行之前，下面每个case都加了判断

qingqing01 · 2022-08-25T04:38:33Z

paddle/fluid/inference/tensorrt/plugin/generic_plugin.cu

+}
+
+template <typename T>
+inline std::string vectorToStr(const std::vector<T>& dims) {


vectorToStr -> VectorToStr

qingqing01 · 2022-08-25T04:40:52Z

paddle/fluid/inference/tensorrt/plugin/generic_plugin.cu

+  outputs_data_type_ = outputs_data_type;
+}
+
+GenericPlugin::GenericPlugin(void const* serialData, size_t serialLength) {


serialData -> serial_data , serialLength -> serial_len

hp03

LGTM

qingqing01 · 2022-09-08T08:12:13Z

paddle/fluid/inference/tensorrt/op_teller.cc

+    if ((desc.HasAttr("namescope") &&
+         PADDLE_GET_CONST(std::string, desc.GetAttr("op_namescope")) ==
+             "/skip_quant_2/") ||
+        desc.HasAttr("skip_quant"))


这里 "skip_quant_2" 太hard code了吧

这部分代码是原有的代码。这里我把这些if判断从 op_conveter.h 文件移动到了 opteller中。所以git 显示是我改的。实际上这里逻辑还是原来的逻辑

qingqing01 · 2022-09-08T08:13:59Z

paddle/fluid/inference/tensorrt/op_teller.cc

+    // only consider dynamic_shape mode
+    if (!with_dynamic_shape) {
+      return false;
+    }


这为啥只考虑 dynamic_shape mode？

qingqing01 · 2022-09-08T08:17:34Z

paddle/fluid/inference/tensorrt/convert/op_converter.h

+        break;
+
+      default:
+        CHECK(false) << "no OpConverter for optype " << op_desc.Type();


输出log信息需要语法正确

qingqing01 · 2022-09-08T08:25:30Z

paddle/fluid/inference/tensorrt/plugin/generic_plugin.cu

+  op_desc_ = std::move(framework::OpDesc(proto_op_desc_, nullptr));
+  // op_meta_data_
+  proto_op_desc_.SerializeToString(&op_meta_data_);
+  // inputs_data_type_ and outputs_data_type_


这些 // 注释看起来无意义吧

qingqing01 · 2022-09-08T08:25:49Z

paddle/fluid/inference/tensorrt/plugin/generic_plugin.cu

+  op_meta_data_ = std::move(op_meta_data);
+  // proto_op_desc_
+  proto_op_desc_.ParseFromString(op_meta_data_);
+  // op_desc_


qingqing01 · 2022-09-08T08:27:55Z

python/paddle/fluid/tests/unittests/ir/inference/test_trt_convert_gather_nd.py

-        yield self.create_inference_config(), (0, 4), 1e-5
-        self.trt_param.precision = paddle_infer.PrecisionType.Half
-        yield self.create_inference_config(), (0, 4), 1e-5
+        yield self.create_inference_config(), (1, 3), 1e-5


这里去掉Half测试的原因是？

已经改回去了

jiweibo · 2022-09-09T06:01:48Z

paddle/fluid/inference/tensorrt/convert/generic_and_custom_plugin_creater.cc

+    for (auto &attr_name : op_attrs_names) {
+      nvinfer1::PluginField plugindata;
+      plugindata.name = attr_name.c_str();
+      if (op_desc.GetAttrType(attr_name) == framework::proto::AttrType::INT) {


貌似可以用宏折叠起来。其他switch case 同

jiweibo · 2022-09-09T06:05:39Z

paddle/fluid/inference/tensorrt/dynamic_shape_infermeta.cc

+namespace inference {
+namespace tensorrt {
+
+nvinfer1::DimsExprs GatherNdInferMeta(


所有的OpInferMeta都要写在这个文件中吗？

嗯集中放在这个文件中

zhangjun · 2022-09-09T07:33:41Z

paddle/fluid/inference/tensorrt/plugin/generic_plugin.cu

+  free(dense_tensor_inputs);
+  free(dense_tensor_outputs);


这里用free可以释放掉吗？

是可以的，只需要释放申请的一个vector对象，vector对象里面的元素，vector在释放的过程中会去释放

析构函数不会被正确调用，有内存泄露风险

已改为delete

zhangjun

LGTM

XiaoguangHu01

LGTM

weishengying closed this Aug 23, 2022

weishengying reopened this Aug 23, 2022

PaddlePaddle locked and limited conversation to collaborators Aug 23, 2022

PaddlePaddle unlocked this conversation Aug 23, 2022

qingqing01 requested review from hp03, jiweibo, zhangjun, qingqing01 and cyj1986 August 25, 2022 04:03

qingqing01 reviewed Aug 25, 2022

View reviewed changes

weishengying closed this Sep 5, 2022

weishengying reopened this Sep 5, 2022

weishengying closed this Sep 5, 2022

weishengying reopened this Sep 5, 2022

weishengying closed this Sep 5, 2022

weishengying reopened this Sep 5, 2022

hp03 reviewed Sep 8, 2022

View reviewed changes

qingqing01 reviewed Sep 8, 2022

View reviewed changes

qingqing01 previously approved these changes Sep 9, 2022

View reviewed changes

weishengying dismissed qingqing01’s stale review via 29af0d2 September 9, 2022 05:11

jiweibo reviewed Sep 9, 2022

View reviewed changes

qingqing01 previously approved these changes Sep 9, 2022

View reviewed changes

zhangjun reviewed Sep 9, 2022

View reviewed changes

weishengying dismissed qingqing01’s stale review via 543f312 September 13, 2022 02:13

paddle-bot-old bot added the contributor External developers label Sep 13, 2022

qingqing01 previously approved these changes Sep 14, 2022

View reviewed changes

zhangjun previously approved these changes Sep 14, 2022

View reviewed changes

XiaoguangHu01 previously approved these changes Sep 14, 2022

View reviewed changes

General Plugin Mechanism

a3a20a4

weishengying dismissed stale reviews from XiaoguangHu01, zhangjun, and qingqing01 via a3a20a4 September 14, 2022 07:12

qingqing01 approved these changes Sep 14, 2022

View reviewed changes

raindrops2sea approved these changes Sep 14, 2022

View reviewed changes

jiweibo merged commit bc77e6d into PaddlePaddle:develop Sep 15, 2022

weishengying mentioned this pull request Sep 15, 2022

General Plugin Mechanism (#45355) #46070

Merged

qingqing01 pushed a commit that referenced this pull request Sep 15, 2022

General Plugin Mechanism (#45355) (#46070)

0793311

paddle-bot-old bot removed the contributor External developers label Sep 15, 2022

weishengying deleted the GenericPluginV1 branch September 18, 2022 15:50

zhangjun mentioned this pull request Nov 22, 2022

[inference][trt] Fp16 support for Generic plugin #48253

Merged

DrRyanHuang mentioned this pull request Nov 23, 2022

[tracking issue] Paddle-TensorRT 算子开发 #48292

Closed

cloud2009 mentioned this pull request Feb 20, 2023

【PaddlePaddle Hackathon 4】核心框架开源贡献 TensorRT 开发任务合集 #50660

Closed

General plugin mechanism #45355

General plugin mechanism #45355

Conversation

weishengying commented Aug 23, 2022 • edited Loading

PR types

PR changes

Describe

通用 Plugin 以及 自定义Op Plugin 加载机制

调整 了OpTell；

测试

1. 测试 custom_plugin_creater

2. 测试 custom_plugin_creater

paddle-bot bot commented Aug 23, 2022

paddle-bot bot commented Aug 23, 2022 • edited Loading

qingqing01 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weishengying Sep 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hp03 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

weishengying Sep 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhangjun left a comment

Choose a reason for hiding this comment

XiaoguangHu01 left a comment

Choose a reason for hiding this comment

weishengying commented Aug 23, 2022 •

edited

Loading

通用 Plugin 以及自定义Op Plugin 加载机制

调整了OpTell；

paddle-bot bot commented Aug 23, 2022 •

edited

Loading

weishengying Sep 8, 2022 •

edited

Loading

weishengying Sep 8, 2022 •

edited

Loading